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Listing of the CLAIMS: 

Without prejudice, this listing of the claims replaces all prior versions and listings of 
the claims in the present application: 

LISTING OF CLAIMS : 

1. (Previously Presented) A foreground object and background sprite separation and 
extraction method for extracting a foreground object and a background sprite, comprising the 
steps of: 

obtaining a global motion for transforming a coordinate system between a reference 
frame and a frame for each of frames in a moving image; 

mapping an original image corresponding to said frame into a reference coordinate 
system for said each of frames by using said global motion, and obtaining a pixel value at a 
point in said reference coordinate system from pixel values of pixels which exist in the same 
point; 

generating a provisional sprite where foreground objects are deleted; 

cutting out a first image from said provisional sprite by using said global motion; 

obtaining a difference image between said first image and said original image; 

extracting, with the provisional sprite, a foreground object image as a region in said 
difference image where each difference value in the region is equal to or higher than a 
threshold, and an other region as a background image; 

mapping said background image to said reference coordinate system by using said 
global motion for said each of frames by inserting a new pixel in a point where a pixel value 
is not yet decided, or by overwriting a pixel, for generating, with the provisional sprite, and 
outputting a background sprite without the foreground objects. 

2. (Original) The foreground object and background sprite separation and extraction 
method as claimed in claim 1, further comprising the steps of: 

cutting out a second image from said background sprite by using said global motion; 
obtaining a difference image between said second image and said original image; 
extracting a foreground object image as a region in said difference image where each 
difference value in the region is equal to or higher than a threshold. 
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3. (Previously Presented) A foreground object and background sprite separation and 
extraction apparatus for extracting a foreground object and a background sprite, comprising: 

means for obtaining a global motion for transforming a coordinate system between a 
reference frame and a frame for each of frames in a moving image; 

means for mapping an original image corresponding to said frame into a reference 
coordinate system for said each of frames by using said global motion, and obtaining a pixel 
value at a point in said reference coordinate system from pixel values of pixels which exist in 
the same point; 

means for generating a provisional sprite where foreground objects are deleted; 
means for cutting out a first image from said provisional sprite by using said global 
motion; 

means for obtaining a difference image between said first image and said original 

image; 

means for extracting, with the provisional sprite, a foreground object image as a 
region in said difference image where each difference value in the region is equal to or higher 
than a threshold, and extracting an other region as a background image; 

means for mapping said background image to said reference coordinate system by 
using said global motion for said each of frames by inserting a new pixel in a point where a 
pixel value is not yet decided, or by overwriting a pixel, for generating, with the provisional 
sprite, and outputting a background sprite without the foreground objects. 

4. (Original) The foreground object and background sprite separation and extraction 
apparatus as claimed in claim 3, further comprising: 

means for cutting out a second image from said background sprite by using said 
global motion; 

means for obtaining a difference image between said second image and said original 

image; 

means for extracting a foreground object image as a region in said difference image 
where each difference value in the region is equal to or higher than a threshold. 
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5. (Previously Presented) A computer readable medium storing program code for 
causing a computer to extract a foreground object and a background sprite, comprising: 

program code means for obtaining a global motion for transforming a coordinate 
system between a reference frame and a frame for each of frames in a moving image; 

program code means for mapping an original image corresponding to said frame into a 
reference coordinate system for said each of frames by using said global motion, and 
obtaining a pixel value at a point in said reference coordinate system from pixel values of 
pixels which exist in the same point; 

program code means for generating a provisional sprite where foreground objects are 
deleted; 

program code means for cutting out a first image from said provisional sprite by using 
said global motion; 

program code means for obtaining a difference image between said first image and 
said original image; 

program code means for extracting, with the provisional sprite, a foreground object 
image as a region in said difference image where each difference value in the region is equal 
to or higher than a threshold, and an other region as a background image; 

program code means for mapping said background image to said reference coordinate 
system by using said global motion for said each of frames by inserting a new pixel in a point 
where a pixel value is not yet decided, or by overwriting a pixel, for generating, with the 
provisional sprite, and outputting a background sprite without the foreground objects. 

6. (Original) The computer readable medium as claimed in claim 5, further 
comprising: 

program code means for cutting out a second image from said background sprite by 
using said global motion; 

program code means for obtaining a difference image between said second image and 
said original image; 

program code means for extracting a foreground object image as a region in said 
difference image where each difference value in the region is equal to or higher than a 
threshold. 
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7. (Original) A segmentation mask extraction method in object coding in moving 
image coding, comprising the steps of: 

receiving a foreground mask image where a foreground part is represented by first 
value and a background part is represented by a second value; 

providing a first value as an alpha value to all shape pixels in each of first 
macro-blocks when the number of pixels of said foreground part in said first macro-block is 
equal to or larger than a first predetermined value n (n^ 1); 

providing said first value as said alpha value to all shape pixels in each of second 
macroblocks when the number of pixels of said foreground part in said second macro-block is 
equal to or larger than a second predetermined value m (m<n), wherein said second 
macro-block is close to said first macro-block where said first value is provided; and 

outputting said segmentation mask. 

8. (Original) The segmentation mask extraction method as claimed in claim 7, further 
comprising the steps of: 

receiving each of third macro-blocks which has been determined as said background 
part; and 

providing said first value to said third macro-block when a difference image between 
a background image and an original image which correspond to said third macro-block 
includes a pixel which has a difference value equal to or larger than a threshold. 

9. (Original) A segmentation mask extraction method in object coding in moving 
image coding, comprising the steps of: 

receiving a foreground mask image; 

generating a number map by calculating the number of pixels of a foreground part for 
each of macro-blocks in said foreground mask image; 
initializing a foreground map; 

providing a predetermined value to each of positions in said foreground map 
corresponding to first macro-blocks when a value of said number map corresponding to said 
first macro-block is equal to or larger than a first predetermined value n (n^l); 
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providing said predetermined value to each of positions in said foreground map 
corresponding to second macro-blocks when a value of said number map corresponding to 
said second macro-block is equal to or larger than a second predetermined value m (m<n) 5 
wherein said second macro-block is close to said first macro-block where said predetermined 
value is provided; and 

generating said segmentation mask from said foreground map and outputting said 
segmentation mask. 

10. (Original) A segmentation mask extraction apparatus in object coding in moving 
image coding, comprising: 

means for receiving a foreground mask image where a foreground part is represented 
by a first value and a background part is represented by a second value; 

first macro-block approximation means for providing a first value as an alpha value to 
all shape pixels in each of first macro-blocks when the number of pixels of said foreground 
part in said first macro-block is equal to or larger than a first predetermined value n (n > 1); 

second macro-block approximation means for providing said first value as said alpha 
value to all shape pixels in each of second macro-blocks when the number of pixels of said 
foreground part in said second macro-block is equal to or larger than a second predetermined 
value m (m<n) wherein said second macro-block is close to said first macroblock where said 
first value is provided in said first macro-block approximation means; and 

means for outputting said segmentation mask. 

1 1 . (Original) The segmentation mask extraction apparatus as claimed in claim 1 0, 
further comprising: 

means for receiving each of third macro-blocks which has been determined a said 
background part; and 

means for providing said first value to said third macro-block when a difference 
image between a background image and an original image which correspond to said third 
macro-block includes a pixel which has a difference value equal to or larger than a threshold. 
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12. (Original) A segmentation mask extraction apparatus in object coding in moving 
image coding, comprising the steps of: 

means for receiving a foreground mask image; 

means for generating a number map by calculating the number of pixels of a 
foreground part for each of macro-blocks in said foreground mask image; 
means for initializing a foreground map; 

means for providing a predetermined value to each of positions in said foreground 
map corresponding to first macro-blocks when a value of said number map corresponding to 
said first macro-block is equal to or larger than a first predetermined value n (n ^ 1); 

means for providing said predetermined value to each of positions in said foreground 
map corresponding to second macro-block when a value of said number map corresponding 
to said second macroblock is equal to or larger than a second predetermined value m (m<n) ? 
wherein said second macro-block is close to said first macro-block where said predetermined 
value is provided; and 

generating said segmentation mask from said foreground map and outputting aid 
segmentation mask. 

13. (Original) A computer readable medium storing program code for causing a 
computer to extract a segmentation mask in object coding in moving image coding, 
comprising: 

program code means for receiving a foreground mask image where a foreground part 
is represented by a first value and a background part is represented by a second value; 

first macro-block approximation program code means for providing a first value as an 
alpha value to all shape pixels in each of first macro-blocks when the number of pixels of 
said foreground part in said first macro-block is equal to or larger than a first predetermined 
value n ^ 1); 

second macro-block approximation program code means for providing said first value 
as said alpha value to all shape pixels in each of second macro-blocks when the number of 
pix is of said foreground part in said second macro-block is equal to or larger than a second 
predetermined value m (m<n), wherein said second macro-block is close to said first 
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macro-block where said first value is provided in said first macro-block approximation 
program code means; and 

program code means for outputting said segmentation mask. 

14. (Original) The computer readable medium as claimed in claim 13, further 
comprising: 

program code means for receiving each of third macro-blocks which has been 
determined as said background part; and 

program code means for providing said first value to said third macro-block when a 
difference image between a background image and an original image which correspond to 
said third macroblock includes a pixel which has a difference value equal to or larger than a 
threshold. 

15. (Original) A computer readable medium storing program code for causing a 
computer to extract a segmentation mask in object coding in moving image coding, 
comprising: 

program code means for receiving a foreground mask image; 

program code means for generating a number map by calculating the number of pixels 
of a foreground part for each of macro-blocks in said foreground mask image; 
program code means for initializing a foreground map; 

program code means for providing a predetermined value to each of positions in said 
foreground map corresponding to first macro-blocks when a value of said number map 
corresponding to said first macro-block is equal to or larger than a first predetermined value n 
(nil); 

program code means for providing said predetermined value to each of positions in 
said foreground map corresponding to second macro-blocks when a value of said number 
map corresponding to said second macro-block is equal to or larger than a second 
predetermined value m (m<n), wherein said second macro-block is close to said first 
macroblock where said predetermined value is provided; and 

program code generating said segmentation mask from said foreground map and 
outputting said segmentation mask. 
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16. (Original) A segmentation mask extraction method for extracting a segmentation 
mask by using a difference image between a background image and an image, comprising the 
steps of: 

obtaining said difference image by calculating an absolute difference between said 
background image and said image fox each pixel; 

initializing an energy map for each macroblock of said difference image; 
calculating energy values for said each macro-block; 
obtaining an average of said energy values; 

calculating a foreground ratio which is a ratio of the size of a foreground mask to the 
size of said image; and 

generating said segmentation mask by using said foreground ratio. 

17. (Previously Presented) A segmentation mask extraction method for extracting a 
segmentation mask by using a difference image between a background image and an image, 
comprising the steps of: 

obtaining said difference image by calculating an absolute difference between said 
background image and said image fox each pixel; 

initializing an energy map for each macroblock of said difference image; 
calculating energy values for said each macro-block; 
obtaining an average of said energy values; 

calculating a foreground ratio which is a ratio of the size of a foreground mask to the 
size of said image; 

generating said segmentation mask by using said foreground ratio; 

obtaining a divided value by dividing said energy value by said average for said each 
macroblock, and providing 0 as an energy value to a macro-block when said divided value is 
equal to or smaller than a (a ^ 1.0); 

obtaining a maximum energy value as a first predetermined value, setting a second 
predetermined value which is smaller than said first predetermined value, and initializing a 
foreground map; 

initializing a temporary foreground map; 



9 



Appl. Ser. No* 09/667,097 
Att. Docket No. 10746/21 

Reply to Final Office Action of August 1 1 , 2004 

providing a predetermined value to each macro-block position in said temporary 
foreground map where said energy value is equal to or larger than said first predetermined 
value; 

counting a count number of macro-blocks where said temporary foreground map has 
said predetermined value; 

generating said segmentation mask from said foreground map and outputting said 
segmentation mask if a value obtained by dividing said count number by the number of all 
macro- locks is larger than a third predetermined value which is predetermined, if not, 
copying values of said temporary foreground map to said foreground map; 

iterating a providing step until a divided number obtained by dividing said count 
number by the number of all macro-blocks becomes larger than said third predetermined 
value, wherein said providing step is a step of providing said predetermined value to each 
macro-block position in said temporary foreground map where said energy value is equal to 
or larger than said second predetermined value, said each macro-block being close to a 
macro-block which has said predetermined value in said foreground map; and 

when said divided number does not become larger than said third predetermined value 
after iterating said providing step, copying values of said temporary foreground map to said 
foreground map, updating said first predetermined value and said second predetermined 
value, and performing said steps after said step of initializing said temporary foreground map. 

18. (Original) A segmentation mask extraction apparatus for extracting a 
segmentation mask by using a difference image between background image and an image, 
comprising: 

means for obtaining said difference image by calculating an absolute difference 
between said background image and said image for each pixel; 

means for initializing an energy map for each macro-block of said difference image; 
means for calculating energy values for said each macro-block; 
means for obtaining an average of said energy values; 

means for calculating a foreground ratio which is a ratio of the size of a foreground 
mask to the size of said image; and 

means for generating said segmentation mask by using said foreground ratio. 
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19. (Previously Presented) A segmentation mask extraction apparatus for extracting a 
segmentation mask by using a difference image between background image and an image, 
comprising: 

means for obtaining said difference image by calculating an absolute difference 
between said background image and said image for each pixel; 

means for initializing an energy map for each macro-block of said difference image; 
means for calculating energy values for said each macro-block; 
means for obtaining an average of said energy values; 

means for calculating a foreground ratio which is a ratio of the size of a foreground 
mask to the size of said image; 

means for generating said segmentation mask by using said foreground ratio; 

means for obtaining a divided value by dividing said energy value by said average for 
said each macro-block, and providing 0 as an energy value to a macro-block when said 
divided value is equal to or smaller than a (a ^ 1 .0); 

means for obtaining a maximum energy value as a first predetermined value, setting a 
second predetermined value which is smaller than said first predetermined value, and 
initializing a foreground map; 

means for initializing a temporary foreground map; 

means for providing a predetermined value to each macro-block position in said 
temporary foreground map where said energy value is equal to or larger than said first 
predetermined value; 

means for counting a count number of macro-blocks where said temporary foreground 
map has said predetermined value; 

means for generating said segmentation mask from said foreground map and 
outputting said segmentation mask if a value obtained by dividing said count number by the 
number of all macro-blocks is larger than a third predetermined value which is 
predetermined, if not, copying values of said temporary foreground map to said foreground 
map; 

means for iterating a providing step until a divided number obtained by dividing said 
count number by the number of all macro-blocks becomes larger than said third 
predetermined value, wherein said providing step is a step of providing said predetermined 
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value to each macro-block position in said temporary foreground map where said energy 
value is equal to or larger than said second predetermined value, said each macro-block being 
close to a macro-block which has said predetermined value in said foreground map; and 

means for copying values of said temporary foreground map to said foreground map, 
updating said first predetermined value and said second predetermined value, and performing 
said steps after said step of initializing said temporary foreground map, when said divided 
number does not become larger than said third predetermined value after iterating said 
providing step. 

20. (Original) A computer readable medium storing program code for causing a 
computer to extract a segmentation mask by using a difference image between a background 
image and an image, comprising: 

program code means for obtaining said difference image by calculating an absolute 
difference between said background image and said image for each pixel; 

program code means for initializing an energy map for each macro-block of said 
difference image; 

program code means for calculating energy values for said each macro-block; 
program code means for obtaining an average of said energy values; 
program code means for calculating a foreground ratio which is a ratio of the size of a 
foreground mask to the size of said image; and 

program code means for generating said segmentation mask by using said foreground 

ratio. 

21. (Previously Presented) A computer readable medium storing program code for 
causing a computer to extract a segmentation mask by using a difference image between a 
background image and an image, comprising: 

program code means for obtaining said difference image by calculating an absolute 
difference between said background image and said image for each pixel; 

program code means for initializing an energy map for each macro-block of said 
difference image; 

program code means for calculating energy values for said each macro-block; 
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program code means for obtaining an average of said energy values; 
program code means for calculating a foreground ratio which is a ratio of the size of a 
foreground mask to the size of said image; 

program code means for generating said segmentation mask by using said foreground 

ratio; 

program code means for obtaining a divided value by dividing said energy value, by 
said average for said each macro-block, and providing 0 as an energy value to a macro-block 
when said divided value is equal to or smaller than a (a ^ 1 .0); 

program code means for obtaining a maximum energy value as a first predetermined 
value, setting a second predetermined value which is smaller than said first predetermined 
value, an initializing a foreground map; 

program code means for initializing a temporary foreground map; 

program code means for providing a predetermined value to each macro-block 
position in said temporary foreground map whey said energy value is equal to or larger than 
said first predetermined value; 

program code means for counting a count number of macro-blocks where said 
temporary foreground map has said predetermined value; 

program code means for generating said segmentation mask from said foreground 
map and outputting said segmentation mask if a value obtained by dividing said count 
number by the number of all macro-blocks is larger than a third predetermined value which is 
predetermined, if not, copying values of said temporary foreground map to said foreground 
map; 

program code means for iterating a providing step until a divided number obtained by 
dividing said count number by the umber of all macro-blocks becomes larger than said third 
predetermined value, wherein said providing step is a step of providing said predetermined 
value to each macro-block position in said temporary foreground map where said energy 
value is equal to or larger than said second predetermined value, said each macro-block being 
close to a macro block which has said predetermined value in said foreground map; and 

program code means for copying values of said temporary foreground map to said 
foreground map, updating said first predetermined value and said second predetermined 
value, and performing said steps after said step of initializing said temporary foreground map, 
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when said divided umber does not become larger than said third predetermined value after 
iterating said providing step. 

22. (Original) A segmentation mask extraction method for extracting a segmentation 
mask by using a difference image between a background image and an image, comprising: 

a first step of regarding each of first macro-blocks as the foreground when an energy 
value of said first macro-block which is obtained by said difference image is equal to or 
larger than a first predetermined value; 

a second step of regarding each of second macro-blocks as the foreground when an 
energy value of said second macro-block is equal to or larger than a second predetermined 
value, said second macro-block being close to a macro-block which is determined as the 
foreground in said first step. 

23. (Original) The segmentation mask extraction method as claimed in claim 22, 
further comprising a step of iterating said second step for predetermined times. 

24. (Original) A segmentation mask extraction method for extracting a segmentation 
mask by using a difference image between a background image and an image, comprising the 
steps of: 

calculating energy value of each macro-block from said difference image and 
calculating an average of said energy values; 

obtaining a divided value by dividing said energy value by said average for said each 
macro-block, and providing 0 as an energy value to a macro-block when said divided value is 
equal to or smaller than a predetermined value; 

regarding each of first macro-blocks as the foreground when said energy value of said 
first macro-block is equal to or larger than a first predetermined value; 

iterating, predetermined times, a step of regarding each of second macro-blocks as the 
foreground when said energy value of said second macro-block is equal to or larger than a 
second predetermined value, said second macro-block being close to said first macro-block 
which is determined as the foreground. 
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25. (Original) A segmentation mask extraction apparatus for extracting a 
segmentation mask by using a difference image between a background image and an image, 
comprising: 

first means for regarding each of first macro-blocks as the foreground when an energy 
value of said first macro-block which is obtained by said difference image is equal to or 
larger than a first predetermined value; 

second means for regarding each of second macro-blocks as the foreground when an 
energy value of said second macro-block is equal to or larger than a second predetermined 
value, said second macro-block being close to a macro-block which is determined as the 
foreground in said first means. 

26. (Original) The segmentation mask extraction apparatus as claimed in claim 25, 
further comprising means for iterating the process by said second means for predetermined 
times. 

27. (Original) A segmentation mask extraction apparatus for extracting a 
segmentation mask by using a difference image between a background image and an image, 
comprising: 

means for calculating energy values of each macro-block from said difference image 
and calculating an average of said energy values; 

means for obtaining a divided value by dividing said energy value by said average for 
said each macro-block, and providing 0 as said energy value to a macro-block when said 
divided value is equal to or smaller than a predetermined value; means for regarding each of 
first macro-blocks as the foreground when said energy value of said first macro-block is equal 
to or larger than a first predetermined value; 

means for iterating, predetermined times, a step of regarding each of second 
macro-blocks as the foreground when said energy value of said second macro-block is equal 
to or larger than a second predetermined value, said second macro-block being close to said 
first macro-block which is determined as the foreground. 



15 



Appl. Ser. No. 09/667,097 
Att. Docket No. 10746/21 

Reply to Final Office Action of August 11, 2004 

28. (Original) A computer readable medium storing program code for causing a 
computer to extract a segmentation mask by using a difference image between a background 
image and an image, comprising: 

first program code means for regarding each of first macro-blocks as the foreground 
when an energy value of said first macro-block which is obtained by said difference image is 
equal to or larger than a first predetermined value; 

second program code mean for regarding each of second macro-blocks as the 
foreground when an energy value of said second macro-block is equal to or larger than a 
second predetermined value, said second macro-block being close to macro-block which is 
determined as the foreground in said first program code means. 

29. (Original) The computer readable medium as claimed in claim 28, further 
comprising program code means for iterating the process by said second program code means 
for predetermined times. 

30. (Original) A computer readable medium storing program code for causing a 
computer to extract a segmentation mask by using a difference image between a background 
image and an image, comprising: 

program code means for calculating energy values of each macro-block from said 
difference image and calculating an average of said energy values; 

program code means for obtaining a divided value by dividing said energy value by 
said average for said each macro-block, and providing 0 as said energy value to a 
macro-block when said divided value is equal to or smaller than predetermined value; 

program code means for regarding each of first macro-blocks as the foreground when 
said energy value of said first macro-block is equal to or larger than a first predetermined 
value; 

program code means for iterating, predetermined times, a step of regarding each of 
second macro-blocks as the foreground when said energy value of said second macro-block is 
equal to or larger than a second predetermined value, said second macro-block being close to 
said first macro-block which is determined as the foreground. 
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31. (Previously Presented) The method of claim 1, wherein the foreground object 
image is automatically extractable without a chroma key, manual processing is not required, 
and outline information of the foreground object is obtainable. 

32. (Previously Presented) The method of claim 3 1 , wherein the background sprite is 
a good-quality or clear background sprite. 

33. (Previously Presented) The method of claim 1, wherein the background sprite is a 
good-quality or clear background sprite. 

34. (Previously Presented) The apparatus of claim 3, wherein the foreground object 
image is automatically extractable without a chroma key, manual processing is not required, 
and outline information of the foreground object is obtainable. 

35. (Previously Presented) The apparatus of claim 34, wherein the background sprite 
is a good-quality or clear background sprite. 

36. (Previously Presented) The apparatus of claim 3, wherein the background sprite is 
a good-quality or clear background sprite. 

37. (Previously Presented) The computer readable medium of claim 5, wherein the 
foreground object image is automatically extractable without a chroma key, manual 
processing is not required, and outline information of the foreground object is obtainable. 

38. (Previously Presented) The computer readable medium of claim 37, wherein the 
background sprite is a good-quality or clear background sprite. 

39. (Previously Presented) The computer readable medium of claim 5, wherein the 
background sprite is a good-quality or clear background sprite. 
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40. (Previously Presented) The method of claim 7, wherein the two-stage macro-block 
approximation for extracting a segmentation mask provides that there is no hole in an 
extracted object. 

4 1 . (Previously Presented) The method of claim 40, wherein the two-stage macro- 
block based shape approximation reduces macro-block shape information and erosion of the 
foreground part. 

42. (Previously Presented) The method of claim 9, wherein the two-stage macro-block 
approximation for extracting a segmentation mask provides that there is no hole in an 
extracted object. 

43. (Previously Presented) The method of claim 42, wherein the two-stage macro- 
block based shape approximation reduces macro-block shape information and erosion of the 
foreground part. 

44. (Previously Presented) The method of claim 7, wherein two-stage the macro-block 
based shape approximation reduces macro-block shape information and erosion of the 
foreground part. 

45. (Previously Presented) The method of claim 9, wherein the two-stage macro-block 
based shape approximation reduces macro-block shape information and erosion of the 
foreground part. 
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